o 


AD-A285  017 


.  /•  * 


^  v.W'vV-U-  '*  :'  'Se'"V' 


■..  -is'  ,  .;■  -i. 


2^  AN  EVALUATION  OF  DEPARTMENT  OF  DEFENSE 
CONTRACTORS'  COST  PERFORMANCE 


• 


,  .•  .1 


THESIS 

John  M.  Pletcher,  Captain,  USAF 
Jane  C.  Young,  Captain,  USAF 

AFIT/GCA/LAS/94S-5 


.  ■■<*:■  •.'.y-i;-  >■  , 

■  '  ■■  ■  >■• . 


i 


J' 


.  k  ■  ■■ 

_  \- 


'p*y 
.4  •> 


TU»  doeuatni  bos  bMS  appiovad 
te  pdfaBc  mIcos*  and  salt;  its 
dbrnlsutioa  is  unlimitsd 


DEPARTMENT  OP  THE  AIR  FORCE 
AIR  UNIVERSITY 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 


Wright-Patterson  Air  Force  Base,  Ohio 


AFIT/GCA/LAS/94S-5 


AN  EVALUATION  OF  DEPARTMENT  OF  DEFENSE 
CONTRACTORS'  COST  PERFORMANCE 

THESIS 

John  M.  Pletcher,  Captain,  USAF 
Jane  C.  Young,  Captain,  USAF 

AFIT/GCA/LAS/94S-5 


Accesion  For 


NTIS  CRA&l 
DTiC  TAB 


U:id:'inoii;.;xa 

Justification 


By . 

Distribution  / 
Avaiiabui 
I  Av.;:! 


Dist 


r.'J ,  or 
•c.ai 


A-/ 


DTIC  QUALTr?  nJSPEGTED  3 


Approved  for  public  release:  Distribution  unlimited 


94  9  28  l)7ji 


^*^94-31130 


n  n 


The  views  expressed  in  this  thesis  are  those  of  the  authors 
and  do  not  reflect  the  official  policy  or  position  of  the 
Department  of  Defense  or  the  U.S.  Government. 


AFIT/GCA/LAS/94S-5 


AN  EVALUATION  OF  DEPARTMENT  OF  DEFENSE 
CONTRACTORS'  COST  PERFORMANCE 

THESIS 

Presented  to  the  Faculty  of  the  School  of 
Logistics  and  A.cquisition  Management 
of  the 

Air  Force  Institute  of  Technology 
Air  University 

In  Partial  Fulfillment  of  the 
Requirements  for  the  Degree  of 
Master  of  Science  in  Cost  Analysis 


John  M.  Pletcher,  B.S. 
Captain,  USAF 


Jane  C.  Young,  B.S. 
Captain,  USAF 


September  1994 


Approved  for  public  release;  distribution  unlimited 


Preface 


The  purpose  of  this  study  was  to  investigate  contractor 
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Abstract 

This  study  examines  the  cause  of  cost  overrun 
recoveries  within  Department  of  Defense  (DoD)  contracts.  In 
this  time  of  extremely  limited  congressional  funding,  it  is 
crucial  the  DoD  avoid  cost  overruns.  Information  provided 
to  contracting  officers  and  contractors  which  would  help 
avoid  cost  overruns  would  prove  extremely  valuable  to  the 
DoD.  This  study  attempts  to  address  this  problem  in  two 
ways:  1)  Determine  the  cause  of  overrun  recoveries;  2) 
determine  whether  a  statistical  difference  in  cost  and 
schedule  performance  exists  among  DoD  contractors. 

Interviews,  document  reviews,  and  a  two-sample  t-test 
were  used  to  analyze  the  contracts  that  recovered  from  early 
cost  overruns.  One-Way  Analysis  of  Variance,  along  with 
normality  tests  and  equality-of-variance  tests  were  used  to 
analyze  the  contractors'  cost  and  schedule  performance. 

The  specific  contract  reviewed  for  contract  recovery 
actions  revealed  no  specific  management  action  that  led  to 
the  recovery.  Additionally,  more  than  300  contracts  across 
49  contractors  revealed  no  significant  statistical 


difference  between  contractors  in  the  areas  of  cost  and 
schedule  performance. 


VM 


AN  EVALUATION  OF  DEPARTMENT  OF  DEFENSE 


CONTRACTORS'  COST  PERFORMANCE 


I .  Introduction 


Manag^ent  Issue 

The  Department  of  Defense  (DoD)  solicits  bids  for 
development  and  procurement  of  major  weapon  systems. 
Companies  compete  for  the  contracts  by  submitting  proposals 
for  the  project.  On  the  basis  of  these  proposals,  the  DoD 
awards  the  contract  and  expects  the  contractor  to  deliver 
the  product  without  exceeding  the  budget  for  the  weapon 
system. 

Unfortunately,  the  military  services  have  suffered 
serious  setbacks  in  the  development  of  new  weapon  systems 
due  to  uncontrollable  cost  overruns.  Ideally,  a  contractor 
should  recover  from  an  early  cost  overrun  (complete  a 
project  on  or  under  budget  despite  experiencing  a  cost 
overrun  early  in  the  contract) ;  however,  the  majority  of 
contractors  have  been  unable  to  recover  from  these  overruns. 
The  DoD  would  like  to  break  the  existing  pattern  and  help 
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contractors  deliver  weapon  systems  without  exceeding  the 
specified  budget  for  the  system. 

In  addition,  a  contractor's  cost  performance 
history  should  weigh  heavily  in  the  award  of  a  contract. 
However,  information  pertaining  to  a  contractor's  past 
performance  has  received  little  attention  in  the  source 
selection  process  (11:18168).  An  analysis  of  DoD 
contractors'  past  performance  will  provide  beneficial 
information  to  government  source  selection  teams. 

Specific  Problem 

The  Department  of  Defense  expects  contractors  to 
recover  from  cost  overruns.  However,  very  few  contractors 
have  successfully  recovered  from  cost  overruns  experienced 
beyond  the  15  percent  completion  point  in  the  contract  (1) . 

Additionally,  the  Federal  Government  is  taking  steps  to 
incorporate  past  performance  of  government  contractors  in 
the  source  selection  process  (11:18168).  To  determine  the 
usefulness  of  this  information,  the  statistical  difference 
between  contractors'  cost  performance  should  be  tested.  If 
no  statistical  difference  exists,  the  selection  of  a 
contractor  can  be  made  without  regard  to  past  contractor 
performance . 
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Investigative  Questions 


1.  In  the  past,  have  any  contractors  recovered  from  early 
cost  overruns? 

a.  Did  the  recovery  from  the  cost  overruns  occur 
because  of  specific  action  taken  by  the  contractor? 

b.  What  action  did  the  contractor  take  to  recover  from 
the  cost  overrun? 

2.  Do  commonalities  exist  between  contractors  that  have 
successfully  recovered  from  cost  overruns? 

3.  Is  there  a  significant  statistical  difference  in  the 
cost  performance  of  DoD  contractors? 

4.  Is  there  a  significant  statistical  difference  in  the 
schedule  performance  of  DoD  contractors? 

Scope/Limi ta tr ons 

The  database  used  in  this  study  is  the  1991  version  of 
the  Defense  Acquisition  Executive  Summary  (DAES)  provided  by 
the  Office  of  the  Under  Secretary  of  Defense  for  Acquisition 
[OUSD(A)]  in  Washington  DC.  Our  study  did  not  include  any 
contracts  outside  of  the  DAES  database.  Contractors  not 
submitting  Contract  Performance  Reports  (CPR)  in  the  early 
and  latter  stages  of  the  contract  do  not  provide  an  adequate 
representation  of  contract  performance.  Additionally,  it 
was  necessary  to  compare  contractor  performance  at  several 
contract  completion  points.  Therefore,  consideration  was 
limited  to  contracts  that  submitted  CPRs  before  the  20^  and 
after  the  85%  complete  point. 
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For  purposes  of  analyzing  contract  recoveries,  the 
study  was  further  limited  to  contracts  which  recovered  from 
at  least  a  5%  cost  overrun  when  the  contract  was  more  than 
15%  complete.  This  limitation  was  imposed  to  eliminate 
insignificant  recoveries  where  no  specific  action  would  have 
been  necessary  to  recover  from  the  cost  overrun.  In  this 
situation,  the  initial  overrun  and  eventual  recovery  could 
have  been  due  to  the  "Earned  Value"  allocation  methods  or 
accounting  practices  utilized  by  the  contractor. 

Beneficial  information  as  to  the  causes  of  recovery 
from  a  cost  overrun  could  be  obtained  from  determining  the 
causes  of  the  cost  overrun  itself.  However,  due  to  time 
constraints,  no  attempt  was  made  to  determine  the  causes  of 
contract  cost  overruns. 

The  following  chapter  provides  a  summary  of  the  current 
literature  in  the  area  of  cost  overruns.  Chapter  Three 
details  the  methodology  used  in  the  research,  while  Chapter 
Four  presents  the  results  of  the  analysis.  The  final 
chapter  provides  conclusions  and  recommendations  for  further 
research. 
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II.  Literature  Review 


In  looking  at  the  causes  and  impacts  of  cost  overruns, 
one  must  first  understand  the  environment  in  which  these 
overruns  occur.  In  an  attempt  to  control  contract  costs 
within  the  volatile  acquisition  environment,  the  government 
implemented  several  contract  management  systems  (21:2). 

Beginning  in  1967,  the  DoD  implemented  a  policy 
requiring  defense  contractors  to  comply  with  Cost/Schedule 
Control  Systems  Criteria  (C/SCSC)  to  ensure  effective 
control  of  contract  performance  and  the  provision  of  timely, 
accurate,  and  verifiable  contract  data  to  the  government 
(3:2).  While  not  a  system,  C/SCSC  define  35  standards 
which,  when  followed,  ensure  contractors  employ  sound 
management  practices  (4:2).  These  criteria  are  mandatory 
for  DoD  Research  and  Development  contracts  in  excess  of  $60 
million  and  production  contracts  in  excess  of  $250  million 
(7: 11B2) . 

DoD  5000. 2M  mandates  contractors  required  to  comply 
with  C/SCSC  report  cost  and  schedule  performance 
periodically  in  the  form  of  a  Contract  Performance  Report 
(CPR)  to  evaluate  current  contract  status  (4:14).  Those 
contracts  exempt  from  C/SCSC  compliance  with  a  value 
exceeding  $5  million  and  duration  greater  than  12  months  are 


ti'pically  required  to  submit  cost  and  schedule  performance 
information  via  Cost/Schedule  Status  Report  (7:20-8). 

Another  important  component  in  assessing  contractor 
performance  is  the  Performance  Measurement  Baseline  (PMB) . 
The  PMB  represents  the  Budgeted  Cost  of  Work  Scheduled 
(BCWS)  at  every  point  throughout  the  life  of  the  contract. 

It  is  the  baseline  against  which  contractor  cost  and 
schedule  performance  is  measured. 

Several  studies  have  been  conducted  in  the  area  of 
C/SCSC.  These  studies  have  primarily  focused  on  two  areas: 
(1)  Estimate  at  Completion  (EAC) ,  the  most  current  estimate 
of  final  contract  cost  based  upon  contractor  performance  to 
date;  and  (2)  the  inadequacies  of  current  DoD  cost  control 
measures.  There  is  a  discernible  absence  of  research  in  the 
area  of  contractor  recoveries  from  cost  overruns. 

In  order  to  determine  the  factors  leading  to 
recoveries,  the  probable  causes  of  cost  overruns  must  first 
be  identified.  The  causes  of  cost  overruns  can  be  separated 
into  two  categories,  those  within  the  control  of  program 
managers  and  those  outside  their  control.  The  most  severe 
and  prevalent  causes  of  cost  overruns  fall  into  the  latter 
category. 
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It  is  generally  accepted  that  the  primary  cause  of  most 
overruns  is  unrealistic  PMBs  (12:10)/  a  problem  partially 
outside  the  control  of  program  managers.  Several  factors 
contribute  to  unrealistic  PMBs.  Contractors  typically 
negotiate  unachievable  contract  costs  to  secure  contract 
award  hoping  future  funding  adjustments  will  alleviate  the 
inevitable  overruns  (12:7).  Once  awarded  the  contract,  the 
contractor  must  either  frontload  the  baseline  or  allocate 
unrealistic  budgets  to  every  work  package.  In  reality,  the 
contractor  is  merely  prolonging  inevitable  budget  problems 
(12:7)  . 

Overly  optimistic  government  cost  estimates,  caused  by 
competition  for  limited  funding,  also  lead  to  unachievable 
PMBs  (10:146).  In  a  1986  study,  the  GAO  found  that  over  901 
of  government  contracts  were  awarded  for  amounts  less  than 
the  government's  own  estimates  (10:168).  In  1979,  the 
director  of  the  Procurement  and  Systems  Acquisition  Division 
of  the  General  Accounting  Office  (GAO)  stated  "The  planning 
estimates  (sent  through  the  DoD  and  on  to  Congress)  are  not 
honest.  I  think  they  are  highly  optimistic  for  a  specific 
purpose,  and  that  is  to  get  the  program  started."  (10:146) 

Significant  technical  problems  and  funding  delays  are 
additional  causes  of  obsolete  baselines  (12:6).  Technical 
problems  on  a  contract  require  additional  time,  resources. 


7 


and  effort  to  resolve.  This  additional  effort  disrupts  the 
normal  contract  flow  and  forces  increased  contract 


cost(12:6).  Funding  delays  inevitably  cause  a  stretching  of 
the  contract  schedule  (12:7).  Stretching  the  schedule  adds 
additional  cost  to  the  contract  and  deems  the  original 
baseline  obsolete  (12:7). 

A  second  cause  of  overruns  beyond  the  control  of 
program  managers  is  the  insistence  by  end-item  users  for 
more  exotic  technology  in  weapon  systems  (20:152). 
Contractors  avoid  identifying  unrealistic  specifications  in 
fear  of  being  eliminated  from  further  consideration  for 
contract  award  (10:147).  In  addition,  contractors  must 
submit  excessively  low  bids  to  guarantee  contract  award, 
resulting  in  underfunded  contracts  to  meet  desired 
specifications  (10:147). 

Program  managers  exhibit  little  control  over  their 
program  in  that  a  large  number  of  staff  people  can  insist 
that -the  program  comply  with  a  myriad  of  special 
requirements  without  regard  for  performance  or  price 
(13:49).  Program  managers  typically  include  these  special 
requirements,  intending  to  secure  the  funding  source  at  a 
later  time  (10:147). 
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There  are  a  limited  number  of  causes  of  cost  overruns 
which  the  program  manager  can  directly  impact.  Perhaps  the 
most  damaging  of  these  causes  is  the  failure  to  heed  early 
warning  signs  of  cost/schedule  problems.  The  cancellation 
of  the  Navy's  A-12  program  is  prime  example  of  the 
catastrophic  consequences  which  can  result  from  the  failure 
to  take  action  (2) .  Since  history  shows  overruns  identified 
later  in  the  program  are  less  likely  to  cause  adverse 
congressional  action  and  more  likely  to  receive  corrective 
funding,  program  managers  are  reluctant  to  identify  problems 
early  (13; 52) . 

An  additional  cause  of  cost  overruns  is  a  lack  of 
technological  expertise  by  the  program  office  staff  (13;51). 
This  lack  of  knowledge  hinders  program  managers'  ability  to 
identify  unreasonably  optimistic  performance  goals 
established  by  contractors  (13:51).  The  AH-56A  Cheyenne 
Attack  Helicopter  offers  a  documented  case  where  the  prime 
contractor  defined  unrealistic  performance  goals  which  went 
unchallenged  by  naive  Army  officials,  resulting  in 
cancellation  of  the  program  due  to  exorbitant  cost  increases 
and  technical  problems  (13:51). 

The  lack  of  documented  research  in  the  area  of  cost 
overrun  recoveries  provides  the  opportunity  to  contribute 
essential  information.  The  goal  of  this  research  is  to 


9 


reduce  dollars  lost  to  cost  overruns  by  providing  contract 
managers  with  proven  procedures  for  recovering  from  these 
overruns . 
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The  purpose  of  this  research  is  to  determine  the  cause 
of  contract  recoveries  from  cost  overruns  and  to  analyze  the 
cost  performance  of  DoD  contractors.  This  chapter  outlines 
the  methods  of  analysis  used  to  conduct  the  research.  The 
research  encompassed  two  main  areas:  (1)  the  analysis  of 
successful  contract  recoveries;  and  (2)  the  analysis  of  DoD 
contractors'  cost  performance. 

The  Database 

The  population  of  interest  is  all  DoD  acquisition 
contracts.  The  data  for  the  study  was  taken  from  the 
Defense  Acquisition  Executive  Summary  (DA£S)  database 
provided  by  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition  [OUSD{A)].  The  DAES  database  contains  cost 
performance  data  for  899  completed  or  near-completed 
acquisition  contracts  covering  the  period  June  1970  to 
October  1992.  It  is  a  collection  of  Cost  Performance 
Reports  (CPR)  submitted  monthly/ quarterly  by  contractors  to 
outline  cost  and  schedule  performance.  The  contracts  in 
the  DAES  database  cover  a  wide  range  of  military  products, 
contract  types,  contract  phases,  and  DoD  components. 


The  Sainple 

Initially,  the  DAES  database  was  pared  down  to 
eliminate  those  contracts  exhibiting  an  Over  Target  Baseline 
(0TB)  condition.  When  an  0TB  exists,  the  cost  variance  is 
no  longer  accurately  represented  in  the  CPR.  Therefore,  it 
was  necessary  to  eliminate  all  0TB  contracts  to  ensure  the 
contracts  in  the  sample  contained  accurate  cost  variances. 

The  sample  was  further  reduced  by  eliminating  all 
contracts  which  did  not  meet  the  following  two  criteria: 

(1)  CPRs  must  have  been  submitted  by  the  20%  completion 
point;  and  (2)  CPRs  must  have  been  submitted  at  or  beyond 
the  85%  completion  point.  This  step  was  necessary  to  ensure 
an  accurate  picture  of  the  contracts'  performances  was 
available  for  the  full  life  of  the  contracts.  The  step  was 
also  necessary  to  allow  for  analysis  of  cost  performance  at 
various  stages  of  contract  completion. 

The  reductions  described  above  resulted  in  a  reduced 
database  of  303  contracts  covering  49  contractors  which  met 
the  basic  criteria.  This  reduced  database  constituted  the 
sample  of  interest  for  the  research  effort.  These  contracts 
were  coded  to  allow  publication  of  the  results  without 
revealing  contractor  identities.  The  code  used  is  a  dual 
number  system  with  the  first  number  signifying  the 
contractor  and  the  second  number  signifying  the  contract. 
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For  example,  Contract  25.8  represents  the  eighth  contract  in 
the  database  for  Contractor  #25.  Since  the  sample  is  of 
sufficient  size  to ’adequately  represent  the  population,  it 
is  assumed  the  results  of  this  study  can  be  generalized 
across  all  DoD  contracts. 

Contract  Recoveries 

A  contract  recovery  occurs  when  a  project  is  completed 
on  or  under  budget  despite  experiencing  a  cost  overrun  early 
in  the  contract.  The  steps  presented  in  the  flow  chart 
below  were  necessary  to  reduce  the  sample  to  those  contracts 
which  recovered  from  early  cost  overruns - 


Figure  1.  Decision  Rules  for  Selecting  "Successful' 
Contracts 
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step  1:  First,  it  was  necessary  to  calculate  the  cost 
variance  (CV)  throughout  the  life  of  the  contract.  The  CV 
throughout  the  contract  life  was  calculated  by  subtracting 
the  Actual  Cost  of  Work  Performed  (ACWP)  from  the  Budgeted 
Cost  of  Work  Performed  (BCWP)  for  each  CPR. 

CV^BCWP-ACWP  (1) 


To  calculate  the  CV  at  contract  completion,  it  was 
necessary  to  first  calculate  the  final  ACWP  ( ACWP. )  .  Final 

ACWP  was  calculated  using  extrapolation  from  the  latest  CPR 
for  the  contract.  The  ACWP  was  divided  by  the  percent 
complete  (PC)  reported  on  the  contract's  final  CPR. 


PC  = 


BCWP 

BAC 


(2) 


ACWP^  = 


ACWP 

PC 


(3) 


Final  CV  iCV.)  was  then  computed  by  subtracting  final 
ACWP  from  the  final  Budget  at  Completion  {BAC.).  The  final 
BAC  is  the  BAC  reported  on  the  final  CPR.  The  final  BAC 
reported  has  been  adjusted  for  all  engineering  change  orders 
and  other  contract  modifications. 

CV^  =  BACj^  -  ACWP^  ( 4 ) 
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step  2:  To  determine  those  contracts  which  recovered 
from  early  cost  overruns,  each  contract  in  the  sample  was 
examined  for  an  overrun  beyond  the  151  complete  point. 

Step  3:  Those  contracts  with  overruns  were  further 
examined  to  determine  whether  a  recovery  was  made  by 
contract  completion.  Of  the  303  contracts  in  the  sample, 
only  30  contracts  recovered  from  early  cost  overruns. 

Step  4:  The  30  "successful"  contracts  were  reduced 
further  by  eliminating  all  contracts  which  did  not 
experience  a  negative  cost  variance  of  at  least  five  percent 
beyond  the  15%  complete  point  of  the  contract.  This  step 
eli'.-inated  contracts  which  may  have  recovered  simply  by 
chance  from  accounting  practices  or  "earned  value" 
allocation  methods.  This  reduction  of  the  sample  left  nine 
contracts  which  recovered  from  at  least  a  five  percent 
overrun  beyond  the  15%  completion  point. 

Step  5:  Of  the  nine  successful  contracts.  Contract 
37.11  was  a  construction  contract  and  deemed  not 
representative  of  the  population.  Therefore,  this  contract 
was  eliminated  from  further  analysis.  This  elimination  left 
eight  contracts  for  further  study. 
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In  an  effort  to  determine  the  cause  of  the  recoveries, 
the  eight  contracts  which  recovered  from  early  cost  overruns 
were  examined  in  terms  of:  (1)  DoD  component  (Army,  Navy, 

Air  Force) ;  (2)  Contract  Type  (Fixed  Price  or  Cost  Plus) ; 

(3)  Contract  Size  (in  $) ;  (4)  Most  Unfavorable  Schedule 
Variance  Percentage  (SV^%);  (5)  Stability  of  the 

Performance  Measurement  Baseline  (PMB) ;  and  (6)  Use  of 
Management  Reserve  (MR) .  This  examination  was  made  in  an 
effort  to  find  similarities  between  the  contracts  thar 
successfully  recovered. 


To  determine  SV.jl,  the  schedule  variance  percentage 
( SV% )  was  calculated  for  each  CPR  of  the  successful 
contracts.  With  SV%  calculated  throughout  the  life  of  the 
eight  successful  contracts,  SVu%  was  obtained  for  each 
contract.  The  schedule  variance  ( SV  )  and  schedule  variance 
percentage  were  calculated  as  follows: 


SV  =  BCWP-BCWS 


(5i 


SVVo  = 


SV 

BCWS 


(6) 


To  further  investigate  the  causes  of  contract 
recoveries,  a  comparison  of  mean  cost  variance  percentages 
was  conducted  between  the  group  of  contracts  recovering  from 
cost  overruns  (CV^%)  and  the  remainder  of  the  database 
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(CVjl).  This  comparison  was  conducted  at  the  25,  50,  75, 
and  100  percent  complete  points  (first,  second,  third,  and 
fourth  quartile)  of  the  contracts.  Since  the  DAES  database 
did  not  contain  CPRs  at  exactly  25,  50,  75,  and  100  percent 
complete,  linear  interpolation  was  used  to  calculate  the 
cost  variances  at  these  milestones-  This  procedure  provided 
usable  results  with  negligible  error. 


CV^%  =  -^ -  (7) 

«R 

Where : 

i  -  an  index  for  all  contracts  which  recovered 
from  an  early  cost  overrun 

=  the  number  of  contracts  which  recovered  from 
early  cost  overruns 


CVs%=-!^ -  (8) 

ns 

Where : 

j  =  an  index  for  all  contracts  in  the  remainder 
of  the  database 

AI5  =  the  number  of  contracts  in  the  remainder  of 
the  database 


A  two  sample  t-test,  using  Satterthwaite' s 
approximation  for  the  degrees  of  freedom,  was  used  for  the 
comparison  of  mean  cost  variance  percentages.  This  test 
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allows  for  an  accurate  comparison  of  means  without  requiring 
equal  variances  among  the  groups  (18:96). 

A  prerequisite  for  using  the  t-test  is  normality  of  the 
data.  However,  the  t-test  is  robust  in  the  presence  of  mild 
departures  from  normality  (8:343)  To  test  for  normality,  a 
Rankit  plot  was  produced  and  the  Wilk-Shapiro  statistic  was 
computed.  The  hypotheses  for  the  two-sample  t-test  were  as 
follows: 

H,:  CV^  =  CV^ 

H,:  Cy^o^CV;% 

test  statistic  (t'caic)  = 


Where : 

CF„%  =  the  mean  cost  variance  percentage  of  those 
contracts  recovering  from  early  cost 
overruns 

mean  cost  variance  percentage  of  the 
remainder  of  the  database 

5^'  =  the  sample  variance  of  the  recovered 
contracts 

s/  =  the  sample  variance  of  the  remainder  of  the 
database 


cv.%-cv,% 


IS 


=  the  number  of  contracts  recovering  from  an 
early  cost  overrun 

Wj  =  the  number  of  contracts  in  the  remainder  of 
the  database 

Satterthwaite' s  approximation  for  degrees  of  freedom  in 
t'cair  is  calculated  as  follows: 


Where : 

Vr  =  -  1 

Vs  =  -  1 

s  " 

with  degrees  of  freedom 

S/  .  , 

Us=-^  With  degrees  of  freedom 

Note:  v'  is  then  rounded  down  to  the  nearest 
integer  to  use  the  t-table 

critical  value  (t.mt)  =  value  from  table  of  cumulative 

probabilities  of  student's  t 
distribution  (two-tailed  test) 

When  it'^airl  >  tc-it  ,  /f-,  is  rejected  indicating  the  mean 
cost  variance  percentages  of  the  recovered  contracts  and  the 
remainder  of  the  database  are  not  equal  at  the  95c 
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confidence  level.  This  hypothesis  test  was  conducted  at  the 
25,  50,  75,  and  100  percent  complete  points  of  the 
contracts.  The  Statistix  4.0  compucer  program  was  used  to 
conduct  .this  test  (19).  The  results  of  the  hypothesis  test 
are  discussed  in  Chapter  Four. 

The  objective  of  the  Wilk-Shapiro  test  is  to  evaluate 
the  normality  of  a  sample  (17:602).  The  Wilk-Shapiro 
statistic  is  an  effective  measure  of  normality  even  for 
small  samples  (n<20)  (17:602).  The  SAS  computer  program  was 

used  to  calculate  the  Wilk-Shapiro  statistic  for  the  25,  50, 
75,  and  100%  complete  points  of  the  contracts.  The  Wilk- 
Shapiro  statistic,  along  with  the  Rankit  plots,  provides  an 
accurate  measure  of  the  normality  of  the  samples. 

The  final  step  in  analyzing  contracts  recovering  from 
cost  overruns  involved  selecting  one  contract  for  more  in- 
depth  analysis.  Due  to  its  product  type,  contract  size, 
magnitude  of  recovery,  and  breadth  of  contractor's  previous 
experience.  Contract  37.4  was  selected  as  the  contract  to  be 
analyzed  further. 

To  conduct  the  analysis,  a  supplementary  form  to  the 
Cost  Performance  Report  (CPR)  was  reviewed  for  each 
reporting  period  of  the  contract.  This  form  provided  cost 
and  schedule  data,  as  well  as  narrative  explanations  of  cost 


20 


and  schedule  variances.  These  reports  were  reviewed 
extensively  to  capture  relevant  contract  information. 

Firstly,  the  information  in  the  DAES  database  was 
compared  for  accuracy  to  the  actual  supplementary  forms 
obtained  from  the  managing  System  Program  Office.  Secondly, 
the  supplementary  forms  covering  the  periods  in  which  the 
contract  was  experiencing  negative  cost  variances  were 
reviewed  in  detail  in  an  effort  to  determine  the  factors 
which  caused  the  overrun.  Finally,  the  supplementary  forms 
covering  the  periods  in  which  the  contractor  was  recovering 
from  the  cost  overruns  were  reviewed  extensively.  The 
narrative  of  the  CPRs  was  studied  to  determine  whether  any 
specific  management  action  was  identified  by  the  contractor 
as  leading  to  the  overrun  or  recovery. 

This  final  review  was  conducted  in  an  effort  to 
determine  the  specific  management  action  which  led  to  the 
recovery.  The  effort  was  focused  on  identifying  those 
events,  as  detailed  earlier  in  the  literature  review,  which 
management  had  control  over  from  those  events  which  occurred 
regardless  of  management  action.  Events  such  as  accounting 
errors  and  overestimation  of  overhead  rates  led  to  early 
cost  overruns.  Avoiding  these  events,  which  are  within  the 
control  of  the  program  office,  would  reduce  the  cost 
overruns  experienced  at  the  program  office. 
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Contractor  Performance 

The  sample  of  303  contracts  was  divided  by  contractor, 
resulting  in  49  separate  groups.  This  separation  was 
accomplished  by  ordering  the  contracts  in  the  database 
alphabetically  by  contractor.  The  next  step  involved 
grouping  each  contractor' s  contracts  to  obtain  a  database 
arranged  by  contractor. 

In  an  effort  to  compare  contractor  cost  performance,  a 
mean  cost  variance  percentage  (CV%)  for  each  contractor  was 
computed  at  the  25,  50,  75,  and  100  percent  complete  points 
(first,  second,  third,  and  fourth  quartiles)  of  the 
contracts.  The  CV%  was  computed  by  averaging  the  cost 
variance  percentages  for  all  contracts  administered  by  the 
contractor.  The  following  formula  was  used  to  compute  the 
mean  cost  variance  percentage  for  each  contractor  (CV^.%): 

CV^%=J^ -  (9) 


Where: 

K  =  the  contractor  number 

i  =  an  index  representing  a  specific  contract 
rif.  =  the  number  of  contracts  for  contractor  K 
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To  compare  contractor  performance,  a  comparison  of  mean 
cost  variance  percentages  was  conducted.  The  One-Way 
Analysis  of  Variance  (ANOVA)  option  in  Statistix  Analytical 
Software  (19:121)  was  used  to  conduct  this  comparison  of 
means.  In  order  to  use  One-Way  ANOVA,  the  following  two 
criteria  must  be  met:  (1)  within  group  variances  must  be 
equal  for  all  groups;  and  (2)  each  group's  CV%s  must  be 
normally  distributed  (15:529). 

Bartlett's  test  for  equality  of  variances  between  the 
groups  was  conducted  to  determine  whether  the  variances  were 
equal.  It  should  be  noted  that  the  F-test  for  equality  of 
means  with  the  fixed  ANOVA  model  is  only  slightly  affected 
when  error  variances  are  unequal  (15:624). 

The  test  for  normality  is  important  for  use  of  the 
ANOVA  model.  However,  the  fixed  ANOVA  model  is  robust 
against  small  departures  from  normality  (15:623).  Rankit 
plots  of  all  contracts  in  the  sample  were  used  at  each 
quartile  to  test  for  normality  among  contractor  mean  CV5s. 
Additionally,  Wilk-Shapiro  statistics  were  computed  to 
provide  a  quantitative  test  for  normality. 

As  a  further  analysis  of  the  difference  between 
contractors,  mean  schedule  variance  percentages  were 
investigated.  Initially,  the  most  unfavorable  schedule 
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variance  percentage  (SV^%)  for  each  contract  was  determined 
via  an  automated  search.  This  information  was  used  to 
calculate  each  contractor' s  mean,  most  unfavorable  schedule 
variance  percentage  {SV^%) .  As  with  the  comparison  of  cost 

variance  percentages,  One-Way  ANCVA  was  used  to  compare 
mean,  most  unfavorable  schedule  variance  percentages  between 
contractors . 

The  tests  for  comparison  of  means  were  conducted  using 
a  microcomputer  based  statistical  package  called  Statistix 
Analytical  Software  (19).  A  confidence  level  of  0.95  was 
used  for  all  hypothesis  tests.  Cost  and  schedule  variance 
calculations  were  accomplished  in  an  Excel  spreadsheet  (14) . 
A  SAS  computer  program  was  used  to  calculate  the  Wilk- 
Shapiro  statistics  for  normality  of  the  samples  (9) . 

This  concludes  the  description  of  the  methodology.  The 
next  chapter  presents  the  analysis  of  contract  recoveries  as 
well  as  the  comparison  of  contractors'  cost  and  schedule 
performance . 
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IV »  Analysis 


Analysis  of  Contract  Recoveries 

The  sample  of  303  contracts  was  analyzed  using  the 
procedures  outlined  in  the  previous  chapter  to  determine 
those  contracts  exhibiting  a  recovery  from  an  early  cost 
overrun.  Of  the  303  contracts  in  the  sample,  only  eight 
(2.64%)  successfully  recovered  from  an  early  cost  overrun. 
These  eight  contracts  are  summarized  in  Table  1. 

TABLE  1 

SUMMARY  OF  CONTRACTS  RECOVERING  FROM 
EARLY  COST  OVERRUNS 


Contract 

Code 

Svc 

K  Type 

Final 
BAC  ($M) 

Most  Unfavorable 
CV%  w/  (PC) 

Final 

CV% 

37.4 

A 

FP 

837 

-5.8 

(29) 

5.2 

22.13 

A 

FP 

110 

-34.4 

(29) 

3.1 

22.14 

A 

FP 

26 

-16.7 

(23) 

3.8 

32.9 

N 

CP 

100 

-31.5 

(73) 

1.0 

43.1 

N 

FP 

212 

-8.7 

(22) 

1.0 

19.3 

AF 

CP 

232 

-13.3 

(42) 

1.0 

14.5 

AF 

FP 

749 

-5.3 

(66) 

2.0 

15.9 

A 

FP 

33 

-7.1 

(42) 

0.0 

Average : 

287.4 

-15.4 

2.1 

Svc  =  DoD  Component 
A  =  Army 
N  =  Navy 
AF  =  Air  Force 
BAC  =  Budget  at  Completion 
CV%  =  Cost  Variance  Percentage 
PC  =  Percent  Complete 


K  Type  =  Contract  Type 

FP  =  Fixed  Price  Contrac 
CP  =  Cost  Plus  Contract 
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Of  the  eight  successful  contracts,  four  are  Army 
contracts  while  there  are  two  contracts  from  both  the  Air 


Force  and  the  Navy.  The  majority  of  contracts  have  a  fixed 
price  arrangement.  The  average  final  BAG  for  the  eight 
successful  contracts  is  $287. 4M  and  the  average  final  cost 
variance  is  2.14%.  Maximum  unfavorable  cost  variances  on 
the  eight  contracts  ranged  from  -5.3%  to  -34.4%.  Maximum 
unfavorable  cost  variance  percentages  on  these  eight 
contracts  were  encountered  as  early  as  the  22  and  as  late  as 
the  73  percent  complete  points. 

The  cost  performance  of  the  eight  successful  contracts 
was  compared  against  the  cost  performance  of  the  remaining 
295  contracts  in  the  sample  to  determine  if  statistically 
significant  differences  existed.  Specifically,  the  average 
cost  variance  percentage  of  the  two  groups  was  compared  at 
the  25,  50,  75,  and  100  percent  complete  points  using  a  two- 
sample  t-test. 

The  Rankit  plots  and  Wilk-Shapiro  statistic  for  the  25, 
50,  75,  and  100  percent  complete  points  are  provided  in 
Figures  2,  3,  4,  and  5. 
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Ordem)  D*U 


Figure  2.  Rankit  Plot  of  First  Quartile 
Cost  Variance  Percentage 


Wilk-Shapiro  0. 9025  303  esses 


Figure  3.  Rankit  Plot  of  Second  Quartile 
Cost  Variance  Percentage 
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Figure  4.  Rankit  Plot  of  Third  Quartile 
Cost  Variance  Percentage 


Figure  5.  Rankit  Plot  of  Fourth  Quartile 
Cost  Variance  Percentage 
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The  Wilk-Shapiro  statistic  provides  a  quantitative 
measure  of  the  normality  of  the  sample.  The  Wilk-Shapiro 
statistics  for  each  quartile,  along  with  the  associated  p- 
values,  are  presented  in  Table  2. 


TABLE  2 

WILK-SHAPIRO  STATISTIC  FOR 
MEAN  COST  VARIANCE  PERCENTAGE  BY  QUARTILE 


Quartile 

Wilk-Shapiro 

p-value 

First 

0.9241 

0.0001 

Second 

0.9025 

0.0001 

Third 

0.8552 

O.COOl 

Fourth 

0.7504 

0.0001 

The  Rankit  p^^ots  and  Wilk-Shapiro  statistics  indicated 
the  samples  were  not  perfectly  normally  distributed.  The 
Wilk-Shapiro  statistics  suggested  the  data  were  non-normal. 
However,  several  outliers,  which  are  known  to  heavily 
influence  the  Wilk-Shapiro  statistic,  were  evident  in  the 
Rankit  plots  (16) .  Despite  the  low  Wilk-Shapiro  statistics, 
the  data  appeared  relatively  normal  in  the  Rankit  plots. 

Since  the  data  appeared  approximately  normal,  use  of 
the  two-sample  t  test  was  appropriate.  Results  of  the 
comparisons  from  the  two-sample  t  tests  are  summarized  in 
Table  3. 
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TABLE  3 

COMPARISON  OF  MEAN  COST  VARIANCE  PERCENTAGE 
FOR  "SUCCESSFUL"  CONTRACTS  AND  REMAINDER  OF  DATABASE 


PC 

Mean 

CVr% 

Std 

Dev 

Mean 

CVs% 

Std 

Dev 

tcaic 

Pooled 

s.e. 

p-value 

25% 

-7.72 

8.91 

-2.55 

9.57 

1.62 

9.56 

0.1472 

50% 

-5.39 

3.00 

-4.95 

11.39 

0.35 

11.26 

0.7308 

75% 

-2.89 

7.76 

-6.94 

12.61 

1.43 

12.52 

0.1908 

100% 

2.15 

1.75 

-13.13 

22.57 

10.52 

22.30 

0.0000 

PC  =  Percent  Complete 

CVr%  =  Cost  Variance  Percentage  for  "Successful"  Contracts 
CVs%  =  Cost  Variance  Percentage  for  Remainder  of  Database 


With  a  -  .05,  the  differences  between  the  means  at  the 
25,  50,  and  75  percent  complete  points  are  statistically 
insignificant.  At  contract  completion,  the  difference 
between  the  means  was  statistically  significant 
(p  <  0.0001).  The  eight  contracts  that  recovered  from  cost 
overruns  performed  significantly  better  in  the  final  quarter 
of  the  contract  than  the  rest  of  the  sample. 


A  plot  of  the  comparison  of  the  mean  cost  variance 
percentages  (CV%)  as  the  contracts  progressed  through  the 
25,  50,  75,  and  100  percent  complete  points  illustrates  the 
opposite  directions  the  CV%s  were  heading  as  the  contracts 
moved  toward  completion.  This  plot  is  shown  in  Figure  6. 
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"Successful"  Contracts  vs.  Remainder  of 
Sample  by  Quartile 


Figure  6  depicts  the  favorable  trend  experienced  by  the 
"successful"  contracts  as  compared  to  the  adverse  trend 
experienced  by  the  remainder  of  the  contracts  in  the  sample. 
More  specifically,  the  "successful"  contracts  experienced  a 
more  unfavorable  cost  variance  percentage  early  in  the 
contract  life  (-7.72%  at  25  percent  complete)  than  the 
remainder  of  the  contracts  in  the  sample  (-2.55%  at  25 
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percent  complete) .  However,  the  "successful"  contracts 
recovered  to  a  favorable  cost  variance  percentage  (2.15%  at 
100  percent  complete)  while  the  remaining  contracts'  cost 
variance  percentages  steadily  declined  (-13.13%  at  100 
percent  complete) . 


As  a  means  of  determining  the  cause  of  the  recoveries, 
the  stability  of  the  Performance  Measurement  Baseline  (PMB) , 
exhibited  by  changes  in  BAG,  for  the  eight  contracts  was 
compared  to  that  of  the  entire  sample.  The  results  of  this 
comparison  are  detailed  in  Table  4. 


TABLE  4 

SUMMARY  OF  BAG  GRANGES  FOR  GONTRAGTS 
REGOVERING  FROM  EARLY  GOST  OVERRUNS 


Gontract 

Initial 

Final 

#  BAG 

%  BAG 

%  BAG  Ghange 

Gode 

BAG  ($M) 

BAG  ($M) 

Ghanges 

Ghange 

due  to  MR 

37.4 

831 

837 

6 

2.41 

0.60 

22.13 

83 

110 

7 

37.35 

12.05 

22.14 

25 

26 

2 

12.00 

12.00 

32.9 

64 

100 

8 

65.63 

6.25 

43.1 

182 

212 

12 

19.78 

10.99 

19.3 

278 

232 

9 

37.41 

19.42 

14.5 

638 

749 

9 

17.71 

0.78 

15.9 

30 

33 

2 

10.00 

6.67 

Mean  %  BAG  Ghange 

for  Sample: 

67.39 

11.39 

BAG  =  Budget  at  Gompletion 
MR  =  Management  Reserve 
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The  average  change  in  BAG  for  the  eight  contracts  is 
25.29%,  as  compared  to  67.39%  for  the  entire  sample, 
indicating  a  significant  difference  in  PMB  stability  between 
successful  contracts  and  the  rest  of  the  sample. 

The  average  change  in  BAG  due  to  the  use  of  Management 
Reserve  (MR)  for  the  successful  contracts  is  8.60%,  as 
compared  to  11.39%  for  the  entire  sample.  This  indicates  no 
significant  difference  between  the  two  groups.  Therefore, 
the  greater  stability  of  the  PMB  seen  in  contracts 
recovering  from  cost  overruns  is  not  due  to  differences  in 
the  use  of  MR. 

In  further  investigating  the  causes  of  recovery,  an  in- 
depth  study  of  Gontract  37.4  was  conducted.  Review  of  the 
supplementary  forms  to  the  Gost  Performance  Reports  revealed 
recovery  was  due  primarily  to  correction  of  accounting 
errors,  changes  in  material  prices,  and  correction  of 
misclassif ied  spares  cost. 

In  one  case  the  supplementary  form  showed  a  $1.7M 
favorable  cost  variance  due  to  changes  in  the  price 
materials.  Another  report  showed  a  $2.9M  favorable  cost 
variance  was  due  primarily  to  recovery  of  earned  value  for 
spares  paid  for  in  a  prior  month.  Several  reports 
identified  a  favorable  variance  resulting  from  application 
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of  rates  and  factors.  The  largest  favorable  cost  variance 
($21.5M)  was  due  to  removal  of  credits  for  spares  from  the 
Major  End  Items  category,  where  they  had  been  erroneously 
placed'.  There  was  no  indication  recovery  resulted  from 
proactive  management  efforts. 

The  analysis  of  the  eight  contracts  recovering  from 
early  cost  overruns  indicates  similar  performance  to  the 
remainder  of  the  sample  through  the  75  percent  complete 
point  with  much  higher  performance  in  the  latter  part  of  the 
contracts.  These  successful  contracts  exhibited  a  more 
stable  PMB.  However,  detailed  analysis  of  one  successful 
contract  failed  to  reveal  any  specific  action  taken  by  the 
contractor  which  led  to  the  recovery. 

Analysis  of  Contractor  Performance 

In  an  effort  to  determine  if  a  statistically 
significant  difference  exists  between  contractors,  One-Way 
Analysis  of  Variance  (ANOVA)  was  employed.  One-Way  ANOVA 
was  used  to  compare  the  mean  cost  variance  percentages 
(CV%)  of  all  contractors  in  the  sample  at  the  25,  50,  75, 
and  100  percent  complete  points  (first,  second,  third,  and 
fourth  quartiles) .  The  CV%  at  each  quartile  for  all 
contractors  is  presented  in  Table  5. 
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TABLE  5 


SUMMARY  OF  CONTRACTOR  COST  VARIANCE 
PERCENTAGE  BY  QUART I LE 


Ktr  Mean  CV%  (by  Quartile) 


Code  First  Second 


1 

3.85 

-0.33 

2 

-2.70 

-2.37 

3 

-0.28 

-0.85 

4 

8.10 

4.85 

5 

-6.74 

-6.52 

6 

-0.05 

-0.58 

7 

2.40 

-1.40 

8 

-18.20 

-17.40 

9 

3.40 

3.80 

10 

-1.13 

-5.58 

11 

1.58 

-5.88 

12 

-5.42 

-13.18 

13 

1.40 

-5.40 

14 

-3.85 

-6.83 

15 

1.26 

0.86 

16 

-2.10 

-1.90 

17 

-16.70 

-18.30 

18 

-7.63 

-8.38 

19 

-2.80 

-1.78 

20 

5.96 

4.76 

21 

0.00 

-3.75 

22 

-9.71 

-13.38 

23 

0.94 

-5.36 

24 

0.83 

-1.02 

25 

-19.30 

-18.40 

26 

-2.90 

-2.40 

27 

1.45 

-0.85 

28 

-4.09 

-2.63 

29 

-1.13 

-1.46 

30 

-11.80 

-30.70 

31 

2.36 

-3.16 

32 

-6.31 

-11.14 

33 

6.00 

1.30 

34 

-8.05 

-12.65 

35 

-10.20 

-18,60 

36 

-2.86 

-3.56 

37 

-1.13 

-0.29 

38 

-0.02 

-3,99 

Third 

Fourth 

#  of  Ks 

-3.10 

-7.25 

4 

-2.53 

-6.00 

3 

-4.00 

-4.25 

4 

5.35 

8.00 

2 

-7.82 

-19.60 

5 

-1.07 

-2.86 

22 

-1.30 

0.00 

1 

-15.20 

-21.00 

1 

1.60 

0.00 

1 

-4.05 

-25.50 

4 

-4.27 

-10.00 

4 

-15.80 

-29.00 

5 

-14.30 

-18.00 

1' 

-6.34 

-14.94 

33 

-2.28 

-7.17 

23 

-0.20 

3.50 

2 

-22.20 

-17.00 

1 

-9.35 

-6.67 

6 

-1.96 

-6.00 

5 

3.54 

2.80 

5 

0.00 

-5.00 

2 

-15.55 

-25.20 

15 

-7.14 

-14.40 

5 

-2.42 

-23.83 

6 

-7.20 

-7.00 

1 

-7.45 

-22.50 

2 

-1.75 

-0.50 

-3.88 

-4.63 

8 

-6.47 

-13.71 

7 

-12.20 

-35.00 

1 

-8.01 

-17.13 

16 

-18.05 

-21.15 

13 

-0.45 

-2.00 

4 

-14.05 

-20.00 

2 

-24.10 

-27.00 

1 

-9.73 

-14.88 

8 

-4.00 

-8.60 

20 

-5.11 

-7.50 

16 
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TABLE  5  (con't) 

SUMMARY  OF  CONTRACTOR  COST  VARIANCE 
PERCENTAGE  BY  QUARTILE 


Ktr  Mean  CV%  (by  Quartile) 


Code 

First 

Second 

39 

0.00 

-13.90 

40 

0.00 

0.00 

41 

-3.56 

-6.36 

42 

-27.00 

-15.90 

43 

-4.53 

-10.13 

44 

-14.50 

-8.25 

45 

-4.70 

-3.60 

46 

-10.38 

-13.27 

47 

-1.30 

-2.00 

48 

-3.90 

-6.43 

49 

-2.20 

0.40 

Average 

Total 

-2.69 

-4.96 

Third 

Fourth 

#  of  Ks 

-2.80 

-20.00 

1 

0.00 

0.00 

1 

-3.96 

-0.60 

5 

-0.75 

-131.00 

2 

-12.51 

-9.29 

7 

-2.20 

-2.00 

2 

-2.60 

-3.00 

T 

X 

-17.70 

-20.50 

14 

-2.40 

-5.00 

1 

-12.67 

-12.57 

7 

2.60 

1.00 

1 

-6.84 

-12 ‘73 

303 


CV%  *  Cost  Variance  Percentage 
Ktr  =  Contractor 
K  =  Contract 


Since  One-Way  ANOVA  requires  equality  of  variances, 
Bartlett's  Test  for  equality  of  variances  was  conducted. 
Contractor  #21  had  only  two  contracts,  both  of  which  had  the 
same  CV%  at  the  25  and  75  percent  complete  points.  This 
caused  Contractor  #21  to  have  a  CV%  variance  of  zero  at  the 
first  and  third  quartiles.  Bartlett's  Test  cannot  be 
conducted  if  any  group  variance  is  near  zero.  Therefore, 
the  equality  of  variance  test  was  conducted  without 
Contractor  #21  at  the  first  and  third  quartiles.  The 
results  of  Bartlett's  test  are  presented  in  Table  6. 
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TABLE  6 


BARTLETT'S  TEST  FOR  EQUALITY  OF  VARIANCE 
BETWEEN  CONTRACTORS 


Quartile 

Bartlett's 

Statistic 

p-value 

First 

57.55 

0.0070 

Second 

94.69 

0.0000 

Third 

93.09 

0.0000 

Fourth 

130.42 

0.0000 

The  p-values  provided  by  Bartlett's  Test  indicate  the 
variances  between  groups  are  not  equal.  However,  One-Way 
ANOVA  is  only  slightly  affected  by  unequal  variances 
(15:624).  Therefore,  One-Way  ANOVA  was  still  the  preferred 
test  for  comparison  of  mean  cost  variance  percentages. 

The  second  requirement  to  use  the  fixed  effects  ANOVA 
model  is  normality  of  the  data.  The  data  used  for 
comparison  of  mean  cost  variance  percentages  between 
contractors  is  the  same  data  evaluated  in  the  first  part  of 
the  thesis,  "Contract  Recoveries".  Therefore,  the  Rankit 
plots  and  Wilk-Shapiro  statistics  presented  previously 
(Figures  2,  3,  4,  5  and  Table  2)  were  used  to  analyze  the 
data  for  comparison  between  contractors. 

Although  the  Wilk-Shapiro  statistics  indicate  the  data 
are  non-normal,  the  Rankit  plots  show  the  data  to  be 
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relatively  normal.  Additionally/  the  fixed  effects  ANOVA 
model  is  robust  against  small  departures  from  normality. 
Therefore,  One-Way  ANOVA  was  determined  to  be  the  best 
method  for  comparison  of  mean  cost  variance  percentages  . 
between  contractors.  The  results  of  the  One-Way  ANOVA  test 
for  comparison  of  mean  cost  variance  percentages  of  the  49 
contractors  at  each  quartile  of  contract  completion  are 
presented  in  Table  7. 


TABLE  7 


COMPARISON  OF  CONTRACTORS' 
MEAN  COST  VARIANCE  PERCENTAGES 


Quartile 

^calc 

p-value 

First 

1.81 

0.0020 

Second 

1.47 

0.0312 

Third 

1.39 

0.0575 

Fourth 

2.40 

0.0000 

With  a  =  .05,  One-Way  ANOVA  indicates  there  are 
significant  statistical  differences  between  contractors  at 
the  25,  50,  and  100  percent  complete  points.  With  a 
p-value  of  0.0575,  there  are  no  statistical  differences 
between  contractor  cost  variance  percentages  at  the  75 
percent  complete  point. 


To  further  investigate  the  differences  in  the  first, 
second,  and  fourth  quartiles,  a  pairwise  comparison  of  means 
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was  conducted  using  Statistix  Analytical  Software  (19) .  The 
pairwise  comparison  of  means  identified  certain  contractors 
which  were  primarily  responsible  for  the  small  p-values. 

The  One-Way  ANOVA  test  was  run  again  without  the  outlying 
contractors.  The  results  of  the  ANOVA  test  are  presented  in 
Table  8 . 


TABLE  8 

COMPARISON  OF  CONTRACTORS'  MEAN  COST  VARIANCE 
PERCENTAGES  WITHOUT  OUTLYING  CONTRACTORS 


Quartile 

Ktr  Removed 

Fcalc 

p-value 

First 

#42 

1.53 

0.0211 

Second 

#30 

1.38 

0.0606 

Fourth 

#42 

1.01 

0.4563 

Ktr  =  Contractor 

As  Table  8  illustrates,  removal  of  a  single  contractor 
at  each  quartile  results  in  a  significant  increase  in  the 
respective  p-value.  In  the  case  of  the  second  and  fourth 
quartiles,  removal  of  a  single  contractor  led  to  acceptance 
of  the  null  hypothesis;  no  significant  statistical 
difference  exists  between  contractors. 

In  addition  to  the  comparison  of  mean  cost  variance 
percentage,  the  mean,  most  unfavorable  schedule  variance 
percentage  {SV^%)  was  compared  across  contractors.  Since 
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One-Way  ANOVA  was  used  to  compare  SV„%,  equality  of  the 
variances  and  normality  of  the  data  had  to  be  tested. 

As  with  the  comparison  of  mean  cost  variance 
percentages,  equality  of  variance  between  contractors' 

SV^%s  was  tested  using  Bartlett's  test.  Since 

Contractor  #21  had  only  two  contracts  with  equal  schedule 
variance  percentages,  Bartlett's  test  was  computed  without 
Contractor  #21.  Bartlett's  test  provided  a  test  statistic 
of  80.88  (p-value  =  0.0000).  However,  One-Way  ANOVA  is  only 
slightly  affected  by  unequal  variances.  Therefore,  One-Way 
ANOVA  was  deemed  appropriate  for  the  comparison  of 
contractors'  SV^%  . 

The  test  for  normality  was  accomplished  with  a  Rankit 
plot  and  the  Wilk-Shapiro  statistic.  The  results  of  the 
normality  tests  are  provided  in  Figure  7. 
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Figure  7.  Rankit  Plot  of  Mean,  Most  Unfavorable 
Schedule  Variance  Percentage 

The  associated  p-value  for  the  mean,  most  unfavorable 
schedule  variance  percentage  is  0.0001.  The  Wilk-Shapiro 
statistic  indicates  the  data  are  non-normal.  However,  the 
Rankit  plot  illustrates  the  data  are  relatively  normal, 
ignoring  the  apparent  outliers.  The  apparent  outliers  also 
have  a  significant  effect  on  the  Wilk-Shapiro  statistic. 

The  fixed  effects  ANOVA  model  is  robust  against  small  « 
departures  from  normality. 

The  initial  ANOVA  results  indicate  a  significant 
statistical  difference  between  contractors'  mean,  most 
unfavorable  schedule  variance  percentage.  However,  a 
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pairwise  comparison  of  means,  using  Bonferroni's  method, 
indicated  Contractors  #22  and  #46  were  primarily  responsible 
for  the  difference  in  contractor  SV^%  .  Therefore,  One-Way 

ANOVA  was  computed  a  second  time  without  the  aforementioned 
contractors.  The  ANOVA  results,  with  and  without 
Contractors  #22  and  #46,  are  presented  in  Table  9. 

TABLE  9 

COMPARISON  OF  CONTRACTORS'  MEAN,  MOST  UNFAVORABLE 


SCHEDULE 

VARIANCE 

PERCENTAGES 

Ktrs  Included 

Fcalc 

p-value 

All 

1.75 

0.0032 

Minus  #22, #46 

1.30 

0.1084 

Ktr  =  Contractor 

As  Table  9  illustrates,  a  significant  statistical 
difference  exists  when  all  contractors  were  compared. 
However,  removal  of  Contractors  #22  and  #46  resulted  in  no 
significant  statistical  difference  among  the  contractors. 

This  concludes  the  presentation  of  the  ^nalysis.  The 
next  chapter  presents  the  conclusions  of  the  thesis  project 
as  well  as  areas  for  further  study. 
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Conclusions  and  Recommendations 


Contract  Recoveries 

The  analysis  of  the  303  contracts  indicated  that 
recovery  from  early  cost  overruns  is  a  rare  occurrence. 
Additionally,  determining  the  cause  of  the  recoveries  did 
not  prove  to  be  a  simple  task.  The  "successful"  contracts 
covered  a  wide  range  of  contract  scenarios  and  an  in-depth 
analysis  of  a  single  "successful"  contract  revealed  little 
information  as  to  the  management  actions  which  might  lead  to 
a  recovery. 

Statistical  testing  indicated  a  significant  statistical 
difference  did  exist  between  the  "successful"  contracts  and 
the  remainder  of  the  contracos  in  the  sample.  The  cost 
variance  percentage  on  "successful"  contracts  steadily 
improved,  while  the  cost  variance  percentage  on  the 
remainder  of  the  contracts  in  the  sample  steadily  declined. 

The  single  concrete  measure  which  indicated  a  possible 
cause  for  recovery  on  the  "successful"  contracts  was  the 
stability  of  the  performance  measurement  baseline  (PMB) . 

The  "successful"  contracts  experienced  a  significantly  more 
stable  PMB  than  the  remaining  contracts.  A  stable  PMB 
permits  more  accurate  planning  and  leads  to  more  effective 
use  of  resources. 


43 


The  actions  which  lead  to  recovery  from  cost  overruns 
are  yet  unknown.  Contractors  recovering  from  cost  overruns 
are  not  following  a  "get  well"  management  plan.  Publication 
of  management  actions  leading  to  recovery  is  currently 
infeasible. 

Contractor  Performance 

The  comparison  of  mean  cost  variance  percentage  between 
the  49  contractors  represented  in  the  DAES  database 
indicated  no  significant  difference  between  contractors. 

With  the  exception  of  one  or  two  contractors,  this  result 
indicates  past  cost  performar.ee  should  not  be  a  significant 
factor  in  source  selection  for  a  new  contract. 

The  comparison  of  mean,  most  unfavorable  schedule 
variance  indicated  no  significant  difference  existed  between 
contractors.  Only  two  of  the  49  contractors  in  the  sample 
experienced  a  mean,  most  unfavorable  schedule  variance 
significantly  different  from  the  rest  of  the  sample. 

Limitations 

As  discussed  earlier,  the  statistical  test  used  for 
comparison  of  means  in  the  "Contract  Recoveries"  section 
required  normality  of  the  data.  Although  the  sample  failed 
the  normality  test,  it  was  assumed  the  data  were 
approximately  normal  and  the  two-sample  t  test  was 
appropriate. 
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The  statistical  test  used  for  comparison  of  means  in 
the  "Contractor  Performance"  section  required  normality  of 
the  data  and  equal  variances  between  groups.  Although  both 
tests 'were  failed,  normality  and  equality  of  variances  was 
assumed  for  reasons  presented  in  Chapter  Four. 

Recommendations  for  Further  Study 

This  study  concluded  recoveries  from  early  cost 
overruns  are  rare  and  potentially  caused  by  actions  outside 
of  the  program  manager's  control.  The  identification  of 
specific  management  action  which  may  lead  to  recovery  from 
cost  overrun  is  difficult. 

This  study  also  concluded  no  significant  difference 
exists  between  contractors  in  terms  of  cost  or  schedule 
performance.  However,  the  study  was  limited  in  scope  to 
mean  cost  variance  percentage  and  mean,  most  unfavorable 
schedule  variance  percentage. 

It  is  recommended  that  further  analysis  of  contractors 
recovering  from  overruns  be  conducted.  In-depth  analysis  of 
several  "successful"  contractors  may  identify  specific 
actions  which  led  to  recovery  from  early  cost  overruns.  It 
is  also  recommended  that  Data  Envelope  Analysis  be  employed 
in  an  effort  to  determine  which  contractors  form  the  leading 
edge  of  the  field  in  terms  of  cost  or  schedule  performance. 
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Appendix:  Definitions 


Actual  Cost  of  Work  Performed  (ACWP) .  The  cost  incurred  and 
recorded  in  accomplishing  the  work  perfomned  within  a  given 
time  period  (6: ll-B-2-1) . 

Apportioned  Effort.  Effort  that  is  not  readily  divisible 
into  work  packages,  but  is  related  proportionately  to 
measured  effort  ( 6 : ll-B-2-1) - 

Authorized  Work.  Effort  that  has  been  definitized  and  is  on 
contract,  plus  that  for  which  definitized  contract  costs 
have  not  been  agreed  to,  but  for  which  written  authorization 
has  been  received  ( 6 : ll-B-2-1) . 

Budget  At  Completion.  The  total  Budgeted  Cost  of  Work 
Scheduled  at  contract  completion  (4:11). 

Budgeted  Cost  of  Work  Performed  (BCWP) .  The  sum  of  the 
budgets  for  completed  work  packages  and  completed  portions 
of  open  work  packages,  plus  the  applicable  portion  of  the 
budgets  for  level-of-effort  and  apportioned  effort  (6: ll-B- 
2-1)  . 

Budgeted  Cost  of  Work  Scheduled  (BCWS) ♦  The  sum  of  budgets 
for  all  work  packages,  planning  packages,  etc.,  scheduled  to 
be  accomplished  (including  in-process  work  packages),  plus 
the  amount  of  level-of-effort  and  apportioned  effort 
scheduled  to  be  accomplished  v/ithin  a  given  time  period 
(6:ll-B-2-l)  . 

Contract  Budget  Base.  The  negotiated  contract  cost  plus  the 
estimated  cost  of  authorized  unpriced  work  ( 6 : ll-B-2-2 ) . 

Cost  Account.  A  management  control  point  at  which  actual 
costs  may  be  accumulated  and  compared  to  budgeted  cost  of 
work  performed.  A  cost  account  is  a  natural  control  point 
for  cost/schedule  planning  and  control  since  it  represents 
the  work  assigned  to  one  responsible  organizational  element 
on  one  contract  work  breakdown  structure  element  (6:ll-B-2- 
2)  . 

Cost  Performance  Report  (CPR) .  A  monthly  summary  of  the 
cost  and  schedule  progress  of  a  major  defense  system  usually 
required  on  all  RDT&E  contracts  with  a  value  of  $60  million 
or  more  and  procurement  contracts  with  a  value  of  $250 
million  or  more  (FY90$)  (7:20-7). 
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Cost/Schedule  Control  Systems  Criteria  (C/SCSC) .  A  set  of 
35  criteria  used  to  provide  contractor  and  government 
program  managers  with  accurate  data  to  monitor  execution  of 
their  program  and  to:  1)  preclude  the  imposition  of  specific 
costs  and  schedule  management  control  systems  by  providing 
uniform  evaluation  criteria  to  ensure  contractor  cost  and 
schedule  management  control  systems  are  adequate;  2)  provide 
an  adequate  basis  for  responsible  decision-making  by  both 
contractor  management  and  DoD  component  personnel;  and 
3)  bring  to  the  attention  of  DoD  contractors,  and  encourage 
them  to  accept  and  install,  management  control  systems  and 
procedures  that  are  most  effective  in  meeting  requirements 
and  controlling  contract  performance  (6:11-6-1). 

Cost/Schedule  Status  Report  (C/SSR) .  A  condensed  version  of 
the  CPR  required  for  contracts  over  12  months  in  duration 
and  judged  not  significant  enough  for  C/SCSC  application. 
Contracts  less  than  $5  million  (FY90$)  are  normally  excluded 
(7:20-8) . 

Cost  Variance  (CV) .  The  difference  between  the  Budgeted 
Cost  of  Work  Performed  and  the  Actual  Cost  of  Work  Performed 
(4:20)  . 

Cost  Variance  Percentage  (CV%) .  The  Cost  Variance  expressed 
as  a  percent  of  BCWP. 

Estimate  at  Completion  (EAC) .  Actual  direct  costs,  plus 
indirect  costs  allocable  to  the  contract,  plus  the  estimate 
of  costs  (direct  and  indirect)  for  authorized  work  remaining 
(5:ll-B-2-2)  . 

Level-of-Ef fort  (LOE) .  Effort  of  a  general  or  supportive 
nature  that  does  not  produce  definite  end  products  (6:11-B- 
2-2)  . 

Management  Reserve  (MR) .  An  amount  of  the  total  allocated 
budget  withheld  for  management  control  purposes,  rather  than 
designated  for  the  accomplishment  of  a  specific  task  or  set 
of  tasks.  It  is  not  part  of  the  performance  measurement 
baseline  ( 6 : ll-B-2-2 ) . 

Over  Target  Baseline  (0TB) .  An  increase  resulting  in  a 
Total  Allocated  Budget  in  excess  of  the  Contract  Budget  Base 
(5:ll-B-2)  . 

Performance  Measurement  Baseline  (PMB) .  The  time-phased 
budget  plan  against  which  contract  performance  is  measured. 
It  is  formed  by  the  budgets  assigned  to  scheduled  cost 


accounts  and  the  applicable  indirect  budgets.  For  future 
effort,  not  planned  to  the  cost  account  level,  the 
performance  measurement  baseline  also  includes  budgets 
assigned  to  higher  level  contract  work  breakdown  structure 
elements  and  undistributed  budgets.  It  equals  the  total 
allocated  budget  less  management  reserve  {6: ll-B-2-3) . 

Schedule  Variance  (SV) .  The  difference  between  the  Budgeted 
Cost  of  Work  Performed  and  the  Budgeted  Cost  of  Work 
Scheduled  (4:20). 

Schedule  Variance  Percentage  (SV%)  .  The  Schedule  Variance 
expressed  as  a  percent  of  BCWS. 

Total  Allocated  Budget.  The  sum  of  all  budgets  allocated  to 
the  contract.  Total  allocated  budget  consists  of  the 
performance  measurement  baseline  and  all  management  reserve. 
The  total  allocated  budget  will  reconcile  directly  to  the 
contract  budget  base.  Any  difference  will  be  documented  as 
to  quantity  and  cause. (6: ll-B-2-3) 

Undistributed  Budget.  Budget  applicable  to  contract  effort 
which  has  not  yet  been  identified  to  contract  work  breakdown 
structure  elements  at,  or  below,  the  lowest  level  of 
reporting  to  the  Government  (6: ll-B-2-3) . 

Work  Breakdown  Structure  (WBS) .  A  product-oriented  family 
tree  division  of  hardware,  software,  services,  and  other 
work  tasks  which  organizes,  defines,  and  graphically 
displays  the  product  to  be  produced  as  well  as  the  work  to 
be  accomplished  to  achieve  the  specified  product  (6:6-B-l). 

Work  Packages.  Detailed  tasks  or  material  items  identified 
by  the  contractor  for  accomplishing  work  required  to 
complete  the  contract  ( 6 : ll-B-2-4 ) . 
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